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Abstract 

We show that the method of separation of variables gives a natural generalisation of 
integral relations for classical special functions of one variable. The approach is illus- 
trated by giving a new proof of the "quadratic" integral relations for the continuous 
g-ultraspherical polynomials. The separating integral operator M expressed in terms 
of the Askey- Wilson operator is studied in detail: apart from writing down the char- 
acteristic ( "separation" ) equations it satisfies, we find its spectrum, eigenfunctions, 
inversion, invariants (invariant (/-difference operators), and give its interpretation as 
a fractional g-integration operator. We also give expansions of the A\ Macdonald 
polynomials into the eigenfunctions of the separating operator M and vice versa. 
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1. Introduction 



One of the most powerful methods for solving the spectral problem of a quantum 
integrable system is that of Separation of Variables (SoV). Given a quantum inte- 
grable system with n degrees of freedom defined by a complete set of n commuting 
Hermitian operators (integrals of motion) Hj acting in some Hilbert space 7i, 



[Hj, H k 



0. 



J, A; = 1,2, 



n . 



one tries to find a linear operator M sending any common eigenvector P x of the 
H 's 

H j P x = h x . j P x , (1.2) 
labelled by the quantum numbers A = {Ai, . . . , A n }, into the product F x 



M:P X I' \ Uh^'Jj) 



1.3) 



of functions f\-j(yj) of one variable each referred to as partial, or separated, or 
factorized functions. Notice that the operator M does not depend on the spectrum 
{h X j}™ =1 (or the quantum numbers A) and is thought of as an intertwiner between 
two representations of the eigenfunctions of the integrals Hj, namely: the initial 
representation and the separated (or factorized) ^-representation. The essence of 
the method is that the original multivariable spectral problem (L2) is transformed 
by such a transformation into a set of (simpler) multiparameter spectral problems, 
each being in one variable, given by the separation equations of the form 



V; 



' d h 



h 



o 



, n . 



:i-4) 



where T>j are usually some differential or finite-difference operators in variable jjj de- 
pending on the spectral parameters h x -k- In the context of the classical Hamiltonian 
mechanics the above construction corresponds precisely to the standard definition 
of SoV in the Hamilton- Jacobi equation. 

For a recent review of the SoV method see |3J| . See also |28|, |29| as for applica- 
tions of SoV to the 3-particle Calogero- Sutherland model and its relativistic (or q-) 
analog, respectively. We would like to stress here that our definition of SoV is not 
restricted by purely coordinate changes of variables, as it is usually (historically) 
supposed (see, for instance, the book [23] and references therein). Instead, we al- 
low the separating transform to be a generic canonical transformation in classical 
mechanics or a linear (integral) operator in quantum case. 

The general multiparameter spectral theory for systems of equations like (|1.4| ) 
including, in particular, questions related to completeness of eigenfunctions, the 
Parseval equality and the like, has been studied extensively JF], [11], 0, |l], |42|, [45 



The book [TJ|] (see also survey article ]IT|) gives a comprehensive treatment of mul- 
tiparameter spectral theory in finite dimensional cases. Various results concerning 



unbounded (e.g. differential) operators are brought together in the monograph 



2 



where the author provides a study of the infinite dimensional case via the theory of 
several commuting operators in Hilbert space [Hj v '\. 

In the present paper we will not concentrate on the multiparameter spectral 
problem because for the particular system we study we do know explicitly the spec- 
trum and the solution of the separation equation. Although, when doing SoV for 
more complicated integrable systems, like Toda lattice or elliptic Calogero-Moser 
problem, we expect the above-mentioned theory will play an important role. 

The main problem of SoV (apart from the study of the separation equation and 
its solution) is to try to describe the separating operator M (and its inversion) in 
the most explicit terms. The eigenvectors Pa in a particular representation become 
special functions or orthogonal polynomials in many variables (for instance, Jack or 
Macdonald polynomials) while f\.j in (|1.3| ) become special functions in one variable. 
In this interpretation the corresponding kernel (nucleus) Ai of the integral operator 
M is some special function in 2n variables and the method of SoV provides us 
with two fundamental integral relations between special functions in one and many 
variables: 

t[fx;i{Vi)= J ••■ J dx 1 ...dx n M{y\x) P x {x u . . . , x n ) (1.5a) 

i=l 

P\(xi, . . . ,x n ) =J ■ ■ -J dyi . . . dy n M(x\y) f[fx-i(yi)- (1.5b) 

i=l 

Here M.{x\y) represents the kernel of the inverse M _1 and, we repeat, both kernels 
do not depend on A. 



The first relation in ( |1.5| ) looks like a "product formula" for the one- variable func- 
tions f\-,j{yj) while the second one gives an integral representation for the special 
function P\(xi, . . . , x n ) of many variables in terms of the special functions, fx-jiyj), 
of one variable. These two types of integral relations are not entirely new in the 
theory of special functions. In particular, when n = 2 both integral relations have 
appeared before in one-variable theory. Below we give a brief review of the litera- 
ture on integral relations/equations (|1.5|) for the special functions and orthogonal 



polynomials in one variable. Let us start with the relations of 2-nd type ( |1.5| b). 
In 1846 Liouville j^Of demonstrated that Lame polynomials satisfy certain non- 



linear integral equations of the form 

f(x) = j jdy x dy 2 M(x\y) f( Vl ) f(y 2 ) . (1.6) 
However, he failed to specify for which of the eight types of Lame polynomials these 



equations were valid. Later on Sleeman EDI showed, using the results obtained by 



Arscott in || (see also the work JjJ and the book ||), that integral equations and 
relations may be constructed which are satisfied by all eight types of Lame polyno- 
mials and Lame functions of the second type. The kernels (nuclei) appearing in all 
these equations are the "potential" Green's functions. These results were further 



generalized in [40] to construction of integral equations and relations satisfied by el- 



lipsoidal wave functions of the first and third kinds, the kernels now being the "free 
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space" Green's functions. The partial differential equations for the kernels M.{x\y) 
as well as the corresponding equality of the form (|1.6|) for these cases were first writ- 
ten down in |7j (see also the book ||) under the names "Second integral theorem" 
and "Second integral equation" . Actually, the very existence of the relation ( |1.6| ) is 
closely connected to the corresponding 2-parameter spectral problem for the Lame 
polynomials/functions and for the ellipsoidal wave functions as was first realized by 
Arscott ||. This idea was generalized further to the case of n-parameter spectral 
problem where one gets the following equation 



n 

f(x) = [■■■Jdy 1 ---dy n M(x\y) ft f(y 3 ) (1.7) 

i=i 

as well as equation of the form 

n r r n 

J] fa) = J ■■■ J d Vl ---dy n M(x\y) [] /(%•) . (1.8) 



i=l j=l 



An abstract formulation (in Hilbert space) of both above-mentioned equations (and 
also the theory of solvability of some operator equations behind it) and the char- 
acteristic (determinantal) equations for the corresponding kernels A4(x\y) can be 
found in the monograph |^l| (Chapter 6 titled "An abstract relation"), see also the 
works |23], [H3J. For the case when f(y) is the ellipsoidal wave function the relations 



( |1.7p and ( |1.8| ) (with n = 2) were derived for the first time by Malurkar |32| and 
Moglich PHI , respectively. 



Let us proceed now to the 1-st type integral relations (|1.5| a). The particular form 
of these relations 

/(3/i)/(3/ 2 )= I dx M(y\x) f(x) (1.9) 



is known in the literature as product formulas for the orthogonal polynomials and 
special functions in one variable. The first example was provided by Gegenbauer 
pTf for his polynomials. This approach was developed further for several other 
classical special functions and orthogonal polynomials [44, 38, [l!| |27|, [l], |36], 35, 20 



PT[ II5L |53| and culminated in ]TB| where the product formulas for angular and radial 



spheroidal wave functions were presented. These formulas generalize the results for 
the Gegenbauer (ultraspherical) polynomials and functions as well as for the Mathieu 
functions. The modern approach to the construction of some of the corresponding 
kernels Ai(y\x) employs a PDE technique based on Riemann's integration method 
to solve a Cauchy problem for an associated hyperbolic differential equation (cf. 



15, 33 



Product formulas are very useful identities associated with a discrete or contin- 
uous function system. Generally, such a formula represents the product of any two 
values of an orthogonal polynomial in terms of a Stieltjes integral which depends 
linearly on the polynomial itself. It is a very important tool in the harmonic anal- 
ysis of orthogonal expansions where it usually defines a convolution product on a 
function space (this space usually becomes a Banach algebra under this operation 
and the convolution product plays the same role as ordinary convolution in Fourier 
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analysis). Notice that the right hand side of a product formula usually defines a 
generalized translation operator and that this in turn can be used to establish an 
appropriate convolution structure on the space and often a hypergroup structure on 
the corresponding interval (cf. []T3] for details). The derivation of new product for- 
mulas yields new convolution structures and hypergroups; moreover it may be used 
to obtain more information about the special functions / themselves, since they are 
usually not given in closed form as soon as they lie in the "land beyond Bessel" || . 

In 1914 Whittaker [|B| conjectured that the Heun differential equation JT^, 
is the simplest equation of Fuchsian type whose solution cannot be represented by 
a contour integral (of a simpler, elementary function); instead the nearest approach 
to such a solution is to find an integral equation satisfied by a solution of the differ- 
ential equation. This is indeed a general situation for the higher special functions 
(cf. ||) where one could not hope to get as many explicit formulas and representa- 
tions as there are for special functions of the hypergeometric type. For instance, the 
technique of series solutions leads to a higher order recursion relation involving at 
least three successive coefficients (for functions of the Heun type). This observation 
applies not only to power series but to series of other forms. Although there is no 
proof that a two-term recursion cannot be obtained. Usually, no integral expres- 
sions for higher special functions are available; instead they satisfy certain integral 
equations like the ones mentioned above. 

In the present paper we aim to demonstrate that the integral equations similar 
to those just discussed appear very naturally in the unified approach of SoV. One 
(among the many) way to introduce a multivariable special function generalizing a 
given one-variable special function is to associate it to a quantum integrable family, 
i.e. to obtain it as a common eigenfunction of a commutative ring of n quantum 
integrals of motion. In such a manner many known functions can be described and 
characterized. Assuming the hypothesis that any quantum integrable system can 
be separated we arrive at an interesting problem of studying various integral oper- 
ators, in an attempt to factorize a particular function or polynomial. That was our 
leading motivation for studying 3-variable Jack polynomials in |28| and 3-variable 
Macdonald polynomials in [F29J which are symmetric orthogonal polynomials asso- 
ciated to the root system A 2 characterized by being common eigenfunctions of the 
corresponding quantum Calogero-Sutherland JH|, (43] and Ruijsenaars [[^J models 
of the trigonometric type. It appears that there are explicit separating operators M 
for both problems which factorize polynomials and which can be explicitly inverted 
giving new integral representations for the related polynomials. 

Here we apply the SoV technique to even simpler 2-variable case of the Ai Mac- 
donald polynomials. In particular, we give a new proof of the "quadratic" integral 
equations for the continuous g-ultraspherical polynomials. The separating integral 
operator M expressed in terms of the Askey- Wilson operator is studied in detail: 
apart from writing down the characteristic ( "separation" ) equations it satisfies, we 
find its spectrum, eigenfunctions, inversion, invariants (invariant g-difference opera- 
tors), and give its interpretation as a fractional g-integration operator. We also give 
expansions of the A\ Macdonald polynomials into the eigenfunctions of the separat- 
ing operator M and vice versa. The structure of the operators M for A\ and A 2 
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cases turns out to be quite similar which indicates an intimate relation between the 
two SoV problems. 

The structure of the paper is as follows. In Section 2 we collect basic informa- 
tion about continuous g-ultraspherical polynomials including the product formula 
in integral form. In Section 3 we give the main properties of the Askey- Wilson type 
integral operator M a p which provides us with an important tool for construction and 
description of the separating operator M in the key Section, 4. In the next Section, 
5, we derive expansions for the A\ Macdonald polynomials in terms of the eigenfunc- 
tions of the separating operator M, as well as the dual expansions. As a bi-product 
we obtain an interesting interpretation of M as a quantum integrable map (discrete- 
time system) and find its invariants (integrals of motion): the g-difference operators 
Ni and N 2 - We provide some concluding remarks in the final Section, 6, and give an 
Appendix where the classical SoV for the trigonometric A\ type Ruijsenaars model 
is performed and a crucial (for the present paper) inter-relation between the A 2 and 
A\ problems is explained in detail. 



2. Continuous g-ultraspherical polynomials 

The main references for the formulas and notations in this Section are [^, ^]. 

Continuous g-ultraspherical polynomials C n (£; j3\q) || is an important class of or- 
thogonal polynomials of basic hypergeometric type within the Askey scheme. They 
can be obtained from the general Askey- Wilson polynomials |L0| through a speci- 
fication of their four parameters, so that C n (£; f3\q) depend only on one parameter 
[3, apart from the degree n and basic parameter q. They are defined through the 
(terminated) 20i basic hypergeometric series 



\P\ H)n And i 

7 r- e 201 



q 13 -a a/TV 2 * 



(P; q)k(P; q) n -k e i( n -2k)e 
t^ (q;q)k(q;q)n-k 



£ = cos 9 . 



(2.1) 



Notice that in the limit q f 1 we get usual Gegenbauer (or ultraspherical) polyno- 
mials C£(£) (see 0, m 



limC n (C; q X \q) = f = Cfa) 



k=0 



k\{n-k)\ 



(2.2) 



Polynomials C n (£;/3\q) are orthogonal polynomials on [—1,1] with the following 
measure (\q\ < 1, \/3\ < 1): 



2tt J-i 



C m &P\q)C n {bf3\q) 



\Q"-"\q)oo 



((3e™;q) c 



(£ 1 Pq^(j^q)n (1-/3) 

W 2 ,q;q)°o (g;g)„ 



(2.3) 
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They satisfy the recurrence relation 

2(1 - f3q n ) £ C n (£; /%) = (1 - q n+1 ) C n+1 & (3\q) + (1 - p 2 q n ~ l ) CUfc; (3\q) (2.4) 
and g-difference equation of the form 

(1 - q) 2 D q [«;(£; q*P\q) D q y(0] + K w{& (3\q) y(£) = , y{£) = C n (£; (3\q) , (2.5) 
where 



and DJ(0 := ™ with 5J(e id ) = f(qie w ) - /(<T^), £ = costf. 
There is a simple generating function for these polynomials: 



(e 2 ^;g) c 



(/3e 2 ^;g) c 



A n = 4g-" +1 (l-g")(l-/5V) (2-6) 



(/fe^z, /3e" ie 2; g) 



oo 



(e* e z, e l6 *,2; g) 



£C n (£;/%) A £ = cosfl. 



(2.7) 



°° n =0 



Finally, let us mention the product formula in integral form (see (8.4.1)-(8.4.2) in 
J2g and work |3§) 

C«(fc /%) C„fo; /%) = (3~ n/2 t K{£, rj, C; /%) C n (C; /%) dC (2.8) 

(g;g)„ J-i 

where (£ = cos 9, rj = cos</>, £ = cos-0) 



(g,/?,/3;g)oc (f3e™,f3e^q) c 



27rC9 2 ;g) 00 ^ 
x w(e # ; /35 e i9+ ^, ph^ 6 '^, ^e ie -^, /^e** - *) . 

Function u>(x; a, 6, c, d) in (|2.9| ) is defined as follows: 

w x; a, 6, c, d := . 

(ax, ax i ,ox,ox , cx, cx , ax, ax i ;g) o 

In the Section 4 we will give a new proof of this product formula. 

Remark 1 It is always possible to give a general expression for the kernel K as 
an infinite series. Indeed, consider a product formula in integral form for some 
polynomials p n (x): 



;2.9) 



(2.10) 



Pn(x) p n (y) = dz K{x, y, z) p n {z) 

J a 

and let the orthogonality relation for the polynomials be given by 

rb 

Pn{x) Pm(x) W(x) dx = k n 5 nm . 



(2-11) 



(2-12) 
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Then, the left hand side p n (x)p n (y) of ( |2.11| ) can be thought of as the expression 
for the Fourier coeficients in the expansion of the function K(x, y, z) into the basis 
of orthogonal polynomials p n {z)- Hence, we always have the formula for the kernel 
K(x, y, z) in terms of the product of 3 polynomials p n (x) p n {y) Pn(z) as a consequence 
of ( FTTD and (|2T2|) : 



K(x,y,z) = w(z) 



Pn(x) Pn{y) Pn{z) 



n=0 



Applying this to the kernel ( |2.9|) and using (|2.3|) we obtain the expansion 



K(Z,ri,C,{3\q) 



27r(/5,/?;g) ooN /r^ 



x 



00 



CU£;/%)C n (77;/%)C n (C;/%). (2.13) 



Remark 2 It was shown in [36| that in the limit q j 1 the product formula p 



goes into the product formula due to Gegenbauer |21 

C*(x)C*(y) /•' 



C*(l) Q A (1) 



r 1 C x (z) 



3.14) 



where 



K(x,y,z) 



1-x 2 -y 2 -z 2 + 2xyz) x ~ 1 

_i 

0, 



5(A,!)[(l-z 2 )(l-y 2 )] A - 5 ' 



1 — x 2 — y 2 — z 2 + 2xyz > , 
1 — x 2 — y 2 — z 2 + 2xyz < . 



Now we describe the relation between the g-ultraspherical polynomials and A\ 
Macdonald polynomials. Let Sym(xi,X2) be the space of symmetric Laurent poly- 
nomials in two variables x±, x 2 . Let us introduce the polynomials P\ G Sym(xi, X2), 

Px{x 1} x 2 ) = x? C X21 { x ^f^-t\q) , *± = (x 1 xf 1 )^ , (2.15) 

[h q)\ 2 i 

depending on real parameters q, t G (0, 1) and indexed by the pairs of integers A = 
{Ai, A2} G Z 2 such that Ai < A2. We use the notation |A| = Ai + A2, A21 = A2 — Ai, 
t = q 9 , g > 0. 

Proposition 1 The polynomials P\(xi,x 2 ) \2.1b\ ) coincide with the standard Mac- 
donald polynomials for the root system A\ . 

Proof. From ( |2.5| )- (|2^ ) it follows that Pa ar e the eigenfunctions 

Hj P x = h X;j P x (2.16) 
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of two finite-difference operators in the space Sym(xi,X2) 

Hi = V 12 Tq jXl + V 21 Tq jX2 , H 2 = Tq jXl T q ^ X2 

where 



Vjk 



t2Xj — t 2Xfc 



and T qjX is the g-shift operator in the variable x 

T q , x f(x) = f(qx) . 
The corresponding eigenvalues h\-j are 

h\-,i =t~^q Xl +t*q x *, h Xi 2 = T 
Introducing the basis of monomial symmetric functions m\ in Sym(xi, X2) 



mx{xi,x 2 ) 



Ai Ao 1 Ao Ai 
■^1 J-2 T J-i J-2 , 

Xi x 2 , 



Ai < A 2 
Ai = A2 



and using the definition ( |2.1|) we observe that Pa expands in terms of m v as 



p \= Yl u *» m » ' 

u-<X 



1*1 — Ai (<?;<?) a 2 - 
The ordering -< is defined here as 



v -< A 



Ai < ^1 < v 2 < A 2 • 



(2.17) 
(2.18) 

(2.19) 

(2.20) 

(2.21) 

(2.22) 
(2.23) 
(2.24) 



From ( |2.1| ) there follows also the normalization condition u\\ = 1. The enlisted 
properties of P\ constitute precisely the definition of Ai Macdonald polynomials 
|3J, Chapter VI). 



.see 



Let us introduce also the separated polynomials f\{y) 

fx(y) 



y Xl 20i 



t Q~^ 21 

t -l q x-x 21 -,Q,t' 2 qy 



b.->zl (q;q) X21 n , a n 

V 1 7I~3 Cx 21 { c os0;t\q) 



y = te 



-2i8 



(*5 ?)Aai 

It follows from ( |2.5| ) and fl2.1| ) that f\(y) satisfies the g-difference equation 

[ t{l - qy) Tq2 <y -t*(t- qy) h x . A T q>y + (t 2 - qy) h x , 2 } f x (y) = 
and expands in y as 



fx(y) = Yl Xk y h 

fe=Ai 



(g,t-V~ Aal ;?)fc-Ai ' 



(2.25) 



(2.26) 



(2.27) 
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In particular, 



X\2 



t 



-A21 



(2.28) 



Remark 3 In a general formula is given for the separated polynomials for any 
root system A n -\. Putting n = 2 in the formula (5.1) from [[HJ one obtains another 
expression for f\(y): 



h(y) = y Xl (y,(i)i-2g 2<Pi 



t 



X21 



(2.29) 



The equivalence of fl2.25| ) and ( |2.29j ) follows from the g-analog of Euler's transfor- 
mation formula [pOp , (1-4.3). 



The formula (|2.8|) can now be interpreted as the one expressing SoV for the pair 
of integrals of motion (2.17) for the quantum integrable system with 2 degrees of 
freedom (which is the A\ type trigonometric Ruijsenaars' system |37j). 

Remark 4 As one can see, the Ai Macdonald polynomials P\(xi,X2) ( 2.15 ) are 
already in the factorized form when written in terms of the variables x±. This cor- 
responds to another (trivial) SoV which is purely coordinate one (local transform): 
(xi,x 2 ) 1 — (x + ,x_), as opposed to the integral separating transform ( |2.8|) . This is 
a simple demonstration of the fact that SoV for a given integrable system (a given 
mult i- variable special function) might be not unique in general. 



3. Operator M a p and its properties 

In this and the next sections we introduce and study the integral operator M% 
performing the separation of variables in the A\ Macdonald polynomials (|2.15|) . 



The operator is most conveniently described in terms of a slightly more general 
integral operator M a p which, in turn, is closely related to the fractional g-integration 
operator I a . 

Our main technical tool is the famous Askey- Wilson integral identity [1^, |2^ 



1 f dx 2(abcd;q) 00 

— / — w(x;a,b,c,d) = (3.1) 

2m _ J x {q.ab.ac^ad.bc^bd^cd-.qjoa 



where w is given by (|2.10|) . The cycle T a b c d depends on complex parameters a, 6, c, 
d and is defined as follows. Let C z ^ r be the counter-clockwise oriented circle with 
the center z and radius r. If max(|a| , |6| , |c| , |d| , \q\) < 1 then T a b c d = Co,i. The 
identity (|3.1|) can be continued analytically for the values of parameters a, b, c, d 
outside the unit circle provided the cycle T abcd is deformed appropriately. In general 
case 

^abcd = Co,l + (C zq k £ — C z -i q -k i£ ) , (3.2) 

z=a,b,c,d k>0 
\z\q k >l 

e being small enough for C z ±i q ±k £ to encircle only one pole of the denominator. 
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Now put 



a -1 a_ -| /3 / \ 

a = yq 2 , b = y q 2 , c = rq 2 , d = r q 2 , (3.3) 



and introduce the notation 

C q (u; x, y) := (uxy, vxy' 1 , vx~ x y, vx~ x y~ x \ g)°o . (3.4) 

The kernel 



, . , (i - ?)(?; ?)L i 2 ; ?)oq ; r » 

Map(r, y\x) = y—^j (3.5) 



defines the integral operator 

(KpfM I ^M aP (r,y\x)f(x), (3.6) 

— til J Ju 

■pry 
1 a/3 

the contour Y r A being obtained from T a b C( i by the substitutions (|3.3j ). Notice that 
in ( |3.5|) -( pl6D we consider x and y as arguments and r as a parameter. 

The following properties of the operator JWTg are proved in f29fl , Appendix B. 

Proposition 2 

1. Let Refl(x) be the space of reflexive (invariant w.r.t. x — > x~ x ) Laurent poly- 
nomials in x. Then M 7 aj3 : Refl(x) — > Refl(?/). 



2. In particular, from the Askey-Wilson identity (\3.1\) it follows immediately that 
M : 1 -> 1. 

3. More generally, consider a Laurent polynomial R C j^ 2 k 1 k 2 i r -' x ) e Refl(x) ; where 

ji,2,h,2 e z> 

x (q 2 rx,q 2 rx~ 1 ;q)k 1 (q 2 r~ 1 x,q 2 r~ 1 x~ 1 ;q)k 2 - (3.7) 



T/ten 

: Rjij2kik 2 ( r i X ) 



a/3 i s (<f; q)h+j 2 



(g a+/3 ; q)j 1+ j 2+kl+k , 2 
x(q^ry; q) jl+kl (q^ry" 1 ; q) h+kl 
x(g^r _1 ?/;g) il+fc2 (q^r~ l y~ l ; q) j2+k2 . (3.8) 



The last formula allows to calculate effectively the action of MTg on any poly- 
nomial from Refl(x) . 
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-a,a+0 > 



4- The inversion of is given by the formula 

the corresponding kernel being 

(1 - q)(q; q) 2 ^ {y 2 , y' 2 \ g)oo C q (q$;r, 



(3.9) 



M r a Jx\y) 



x 



2B n (—a, a 



q *;y,x 



£ q [q 2 - r ,y 



(3.10) 



5. The operator M r a Q simplifies drastically when either of the parameters a, f3 
takes negative integer values. Let a = —g, g G Z> . Then M r a p turns into the 
q-difference operator of order g: 



(3-11) 



fc=0 



where 



<- i \ i i\a fc(fc ~ 1} 
Zk(r,y) = 2 



9 
k 



x 



y- 2k (1 - q^y~ 2 ) 

Q 

(q 2 ry,q 2 r l y;q) k {q * ry \q 2 r L y u ,q) g - k 



{qP-9; q) g {q~ k y- 2 ] q) 



(3.12) 



9+1 



It was mentioned briefly in |29j that, similarly to the nonrelativistic case [pq| , 
the operator MIg has a natural interpretation in terms of a fractional q-integration 
operator. To clarify this remark, take the formula (|3.5|) for the kernel Ai a p(r,y\x) 
and rewrite it in the form 



^~\y) 



M a p(r,y\x) = ^rf^ I a (r,y\x), (3.13) 



with 

C q (p\r. .r 



where the kernel T a (r, y\x) is defined as follows: 

<ra . . , (^-q)(q;q)l (x 2 ,x- 2 ;q) oc c q (q^,r,y) 

2T q (a) C q [q2;y,xj C q [q2y 



r, x 



In the operator form we have 

1 

/ Q 
V'r 



K^^T^T^^^- 1 (3.16) 
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where i\f r are thought of as multiplication operators and the operator I a corresponds 
to the kernel Z Q (r, y\x) ( |3.15|) 

r : f(x) -> -L / — X Q (r,y|x) /(*) (3.17) 

Z7TZ J X 



r r -v 



with r considered as a parameter. 

The following properties of the operator I a can be obtained mostly from those 
of Mlg enlisted in the Proposition |2] using the correspondence ( |3.16| ). 

Proposition 3 

1. The operator I a possesses the group property with respect to the parameter a 

2. Define an analog of the power function by the formula (\S.14 )- Then 



r:Vr{x)^r r +a {y). (3.19) 

3. For a = —g, g G Z> the operator I a turns into the finite-difference operator 

J-« : f(x) -> £ C,,fc(r, 2/) /(?*-* y) , (3.20) 

fc=0 

y- 2fe (1 - qs- 2k y- 2 ) C q (q5;r,y) 



C 9 , k (r,y) = (-l) k q 



fc(fc-i) 



q (1 - (q~ k y- 2 ; q) g+1 £ q ( q h ; r , gMy) ' 



(3.21) 

Moreover 

/-f = (3.22) 

if /i ere 

j-i ;/(g) , gjMl!^I f y 2 /(«r*y) i (3 23) 



4- In the limit q | 1 we have 



^--tA^' ( 3 - 24 ) 



rM ^%l~p\ (3^25) 
so, /or instance, equality (\3.19J for a = — 1 gwes t/ie formula 

r 1 ^ = i) v - x . (3.26) 
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The identity ( |3.18| ), like the equivalent identity MIg : 1 — > 1, is a disguised form 
of the Askey- Wilson (AW) integral ([3.1|). The formula ( |3.19|) is also proved quite 
directly using AW integral. The formulas (|3.20|) and (|3.21| ) are equivalent, respec- 



tively, to ( |3.11| ) and ( |3.12| ). The equality ( |3.22j ) is easily proved by induction. 



The properties of the operator I a described in the Proposition [3] allow to consider 
it as a fractional power of the first order difference operator J -1 ( p.23[ ). In the limit 
q f 1 the operator J -1 (|3.24j) is reduced to the pure differentiation operator by the 



change of variable £ = y + y — 2. The operator J Q , respectively, is reduced to the 
well-known Riemann-Liouville-Weyl fractional integral 

r : f(x) - fdy {V " X \ ' f(y) . (3.27) 

In the general case q ^ 1, however, there is no change of variable reducing J -1 ( p.23j ) 
to the pure g-derivative (V q f)(x) = (f(x) — f(qx))/((l — q)x), and our operator 
I a ( |3.17| ) cannot be reduced, respectively, to the fractional g-integration operator 
studied in 0, [3|, |J. It is not our task to develop here the complete theory of the 
operator I a ( |3.17|) although we believe that this new g-fractional integration operator 
deserves more comprehensive study. 



4. Separating operator 

Let us specify the values of the arguments and parameters in the kernel J\4 a p(r, y\x) 



fl3j) to be 

<* = P = 9, y = V-, x = x-, r = t~ 1 y + , y± = (yiy^f 2 , (4.1) 

(in contrast to a = g, (3 = 2g, as for A 2 case 1^9(1 ), and denote the resulting kernel 
as M(y+,y-\x-): 

M(y+,y-\xJ) - ( 1 -^^)lo(f-^ x - 2 ^)°o c ^y-^~ 1 y+) . ( 4 2) 

Assuming £ to be an arbitrary complex parameter, we introduce the integral operator 
acting on functions f{x\,X2) by the formula 

(MJ)( yi ,y 2 ) = -L / — M(y+,y-\z) /(*" kv+z, ■ (4-3) 

Ztti J z 



„t 1 y + ,y- 

1 3,3 



It is apparent from (|4.3| ) that acts on x + as a simple scaling: 

M c : <p( Xl x 2 ) -> vit^^ym) ■ (4.4) 

Now we can formulate the main result of the paper. 
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Theorem 1 The operator (\4-% ), (\j- 6 A ) performs the factorisation of ( or, in other 
words, the SoV for) the A± Macdonald polynomials (\2.1$) : 



M i : P x {x 1 ,x 2 ) -> F x (y l ,y 2 ) = c A>? f x (yi) h{y-z) 



(4.5) 



where the factorized (or separated) polynomial f\(y) is given in (\2.2b\ ) and the nor- 
malization coefficient c\^ is 



-2A1+A2 t|A 



2 ;9)a 21 ' 



(4.6) 



Note that the product formula ( |2.8| ) for the g-ultraspherical polynomials is equiv- 
alent to the formula (4J3). To observe it, it is sufficient to make substitutions t = j3, 
^21 = n, z = e 1 ^ , yi = te~ 2td , y 2 = te~ 2l< ^ and to replace the integral over the unit 
circle by the integral from —1 to 1. 

Taking into account the inversion formulas ( p.9| ) and ( |3.10| ) for Mfg we notice 
that the kernel of the inverse operator M^ 1 is obtained from .M aj g(r, y\x) ( |3.5|) by 
the substitution a = —g, (3 = 2g, x := ?/_, y := x_, r := t^^^ 1 



M(x + ,x-\y_) 



[i-Q)(q;q)l> (y-^y- 2 ^)^ c q (t^,x_,t ^ h 



2B q (-g,2g) 



£q(t;y-,t ^ 1 x, 



(4.7) 



As the immediate corollary the above remark and the Theorem [I| we obtain the 
following result. 



Theorem 2 The inversion of the operator j \4-3j) is given by the formula 



(M^f)( Xl ,x 2 ) 



2ni 



dz 



-9,2s 



M(x + , x-\z) f(U£ 1 x + z,U£ l x + z l ) . 



The operator 1 provides an integral representation for the A\ Macdonald polyno- 
mials in terms of the factorized polynomials f\(y) 



Mr 1 : c A ,£ fx (yi) fx (1/2) - Px(x u x 2 ) . 



(4.9) 



In contrast to the formula (|4.5|) which paraphrases an already known result, the 
formula ( |4.9D leads to a new integral relation for the g-ultraspherical polynomials. 
Note that for positive integer g the operator M^ 1 becomes a g-difference operator 
(cf. Proposition 0:5). 

The proof of the Theorem [I] mimics that in |29| for the A 2 case and consists of 
the following steps. 



Proposition 4 The operator maps bijectively Sym(xi,x 2 ) onto Sjm(yi,y 2 ) 
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Proposition 5 The following operator identity ( "quantum characteristic equation", 
cf. fiEQJ ) is true 



1 - q yj ) 7> M € - £ (1 - qt^yj) T qtV . M ( H, + t(l - qt^yj) M ? # 2 = , (4.10) 



where Hi and H 2 are quantum integrals of motion given in ( \2. 1 % ) 



Proposition 6 The separated polynomial f\(y) \2.2h\ ) is the only Laurent polyno- 
mial solution to the separated equation ( \2.20j) . 



Proposition 7 The normalization coefficient c\ £ in ( U-W is given by the formula 

(M)- 

The main statement of the Theorem [l] follows then by a standard argument (cf. 
ZD]). The Propositions ^ and [S] imply that M^P\ is a Laurent polynomial in y\ and 



y 2 satisfying the separated equation fl2.26| ) in both variables yi^. Then, by virtue 
of the Proposition^, it is factorized into f\(yi) fx (1/2)- It remains only to calculate 
the normalization coefficient c\^ which is done by the Proposition |7|. 

To prove the Proposition |] we shall calculate explicitly the matrix of Mg in special 
bases in Sym(xi, x 2 ) and SymQ/i, y 2 ). Define the bases p v and r u in Sym(xi, x 2 ) and, 
respectively, p v and r„ in Sym(?/i, y 2 ) for v = \v\ < u 2 } G Z 2 : 

Pv = (xiX 2 ) Vl (i~ l Xi^~ 1 X2]q)v 21 , r„ = {x 1 x 2 f' l {tix^ 1 ,tix 2 l \q) V21 , (4.11) 

Pv = (ym) vi (yi,y2-:q)u 21 , r„ = (ym^ify^^y^q)^ ■ (4.12) 
Note that 

p v = (x 1 x 2 )^^ 21 o(^ 1/2 r 1 ^ + ,x_) , r v = (x^rR^Jt-^CW,*-) 

(4.13) 



where FC^k^ ls defined by ( [3.71) . Then, using ( [T8) , we obtain explicit formulas 



for the action of Mg on the bases p u , r u : 

M r . Pu ^^p u , Af e : r„ - //M r„ (4.14) 

where 

,.{p) _ +-v lf: 2ui (A q)f2i (r) _ ,-v 2 £2u 2 



The invertibility of Mg being obvious from (|4.14T ), the Proposition [|is thus proved 



Actually, we have obtained much stronger result, having found an effective way of 
calculating the action of on any polynomial. 

Remark 5 There are involutions and V in Sym(xi, x 2 ) and Sym(|/i, y 2 ), respec- 
tively, 

Uz : <p( Xl , x 2 ) -> ^(t^^r 1 , ^2 X ) , V : y 2 ) -> ^V", ^l/J 1 ) , (4-16) 
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which are intertwined by the operator M^. 

\LJ\. V Mc 



v = {-z/ 2 , -v{\ 



(4.17) 



(4.18) 



Remark 6 One can identify Sym(x 1 ,X2) and Sym (2/1,2/2) setting y^ = £~ 1 Xj. Then 
p v = p v becomes the eigenbasis for Mg. Similarly, setting yj = t£~ x Xj one obtains 
r v = r v as the eigenbasis for M^. 

For a more detailed discussion of the bases p u , r v see the next Section. 
Now we shall prove the Proposition |5|. 

Proof. In [29 1 the proof of the identity analogous to (^4.10 ) was based on shift 
relations for the kernel M.. Here we choose another strategy and prove ( |4 . 1 0| ) purely 
algebraically. 

Note first that the Hamiltonians Hj ( |2.17| ) are described by Jacobian matrices 
in the basis r v (not true for p v \) 



J 'V\,Vx — \ 1 



J 



1,2, 



(4.19) 



where 

K 

c' 



t~2 + U q h 
-t~lq Ul ( 



<-9 



-gH (1 

t 2 C 2 q 2u2 ~ 2 (1 - q U21 ) ■ 



<f 21 ), 



(4.20) 



Now apply the left-hand-side of ( [4.10|) to the basis r„. Using the relations 
r, 1+V2 = r„ (1 - q^-H 2 yl x Y\\ - ^iV)" 1 , 
r vx ^ = r v yr 1 2/ 2 - 1 (l - q^H^ 1 )^! - q^-H 2 y 2 v y\ 



q^jl-q-Hiy- 1 ] 

1 _ q U 21 -l t 2 y -l 



(4.21a) 
(4.21b) 

(4.22) 



one can express all the r's in terms of r v . Discarding the common multiplier r l 
we obtain a rather clumsy rational expression in 2/1, 2/2, t, £, q and q U21 which 
nevertheless, can be shown to be identically zero by a direct computation. 1 



The Proposition y is a direct corollary of the Proposition 12 from [29 |. 

To complete the proof of the Theorem |1] it remains to prove the Proposition [7|. 
The proof repeats, with appropriate adjustments, the proof of the Proposition 7 
from [T2IJ . 

Proof. Using the variables x± and comparing the asymptotics at x_ — > 00 of the 
monomial symmetric function m\ (|2.21 ) 



\M A21 
nix ~ x+ x_, 



x. 



•X- 



(4.23) 
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and of the polynomial r v ( 4.11[) 



~ (_i)^i ? ^2i(^i-i)^2i^2i jjlji* ; x _^oo (4.24) 

we conclude that the expansion of in is triangular 

_l)*2i ? -iA 21 (A 21 -i) r A 21 £-A 21 ^ + (...) r „ , (4.25) 



^A 
y^A 



and, consequently, the expansion of Pa in t v has the same structure. Then using 
(|4.14j )- fl4.15|) and the asymptotics of f v 

r v ~ (-1)^^1^21(^-1)^ y H y vi^ y _ _^ ^ ^ 4 26 ) 

we obtain 

M 5 P x ~ t- A -^ |A| y? 1 y* 21 , 2/- - 00 . (4.27) 



On the other hand, the right hand side of Q4.5|) has the asymptotics 

cx,t fx(yi) hfa) ^ c x ^XXiXx 2 y+ l V- 1 , y-^00. (4.28) 



Comparing ( |4.27| ) and ( |4.28| ), and using formula ( |2.28| ) we obtain the desired ex- 



pression (|4.6|) for c\ 



5. Bases p u and r v 

In this section we establish some more properties of the polynomial bases p u , r u , 
p u , r u introduced by ( ^.11| ), ( |4.12D in the previous Section. Being the eigenfunctions 



of the separating integral operator (see Remark H), these bases must play an 
important role in SoV. 

Our first observation is that the bases can be interpreted also as eigenfunctions 
of certain quantum integrable systems. The following statement is easily verified by 
a direct calculation. 

Proposition 8 The Laurent polynomials p v , p v , r v , r v are the eigenfunctions, re- 
spectively, of the commuting pairs of q- difference operators Nj, Nj, Qj, Qj (j=l,2) 

Nj Pu = q U] Pu , Nj p u = q Uj p u , Qj r v = q~ U] r v , Qj f v = q~ Uj r v , (5.1) 



AT m^-nii) xi{£-x 2 ) Z-xi e,-x 2 . 
jVl — 1 q,x 1 H -Lq,x 2 5 iV 2 — 1 q,xi H 1 q,x 2 > 1,0. Zaj 

£(xi— X 2 ) Hxi— X 2 ) X\—X2 XI— X2 



18 



iVi H i 9iy2 , iV 2 l q , yi H i g,J/2 > (.O.Zbj 

2/l~2/2 2/1-2/2 2/1 -2/2 2/1-2/2 

vi- _ — ; J ^i + r; : J g,*2> ^2- J g,*i + i 5 -^ 

i£(:ci— x 2 ) t?(xi— £2) 21—2:2 xi— u 2 



PC 2/2(1 -yij , y U r -y 2 ; 1 1 -J/l j c -2/2 1 , 

VI - -T- : J Mft + X : J g,2/ 2 ' V2- J g,i/i + J 9,2/2 • l 5 " 2c U 

* (2/1-2/2) f (2/1-2/2) 2/1-J/2 2/1-2/2 

Notice that the above spectral problems are already separated in the variables 

The integral operator Mg obviously intertwines these pairs of first order diffe- 
rence operators 

M e Nj = Nj M i , M c Qj = Qj M { . (5.3) 
Note also that, by virtue of ( |4.1(j| ), 

E/ € N j U^ 1 = Q 3 -i , V ^ F- 1 = Q 3 _, . (5.4) 

Remark 7 Choosing the identification yj = £,~ 1 Xj of the spaces Sym(xi,X2) and 
Sym(?/ 1 , y 2 ), see Remark |6], one can identify Nj = Nj. By virtue of ( |5.3| ) the operator 
commutes then with Nj = Nj and can be considered as the time-shift operator 
for the discrete-time quantum integrable system with integrals of motion N\ and N 2 . 
Same is true for another identification yj = t^Xj and the operators Qj = Qj. 

Having defined several polynomial bases in Sym(2i, x 2 ), such as P\, p u , r u , it is 
natural to calculate the transition matrices between them. The solution is given by 
the following Proposition. 

Proposition 9 The expansion of P\ (\2.15 ) in the bases p u (r u ) ( 4-1 A) ^ s given by 

Px = y E<p» = y E& r * ( 5 - 5 ) 



u^X u^X 



where 



7Tx 



(-i)^e |A| - 2i/1 9^ (H - 2A2+1) (*;g)A2-,i(*;?),2-Ai(g;?)A2i 



(9; q) X 2 -u 2 (t;q)„ 21 {t;q)x 21 {q; q) 



(5.6a) 



"21 



pV = (-l)^(tQ-^ + l A lg^^ +1 -IH)(t; g) U2 _ Xi (t- g ) Aa _^(g; q) Ml ^ 

A (q;q)x 2 -u 2 (q;q)u 1 -x 1 (t;q) U21 (t;q)x 21 (q;q)v 21 

The dual expansions of the bases p\ and r\ ( U-1A) ^ n terms of the A\ Macdonald 
polynomials P u (\2.1b\ ) are as follows: 

Px=Y,Qx P ^ r x = J2RxPu (5.7) 

v^X v^X 

where 

QX ~ (q; q)x 2 -, 2 (q; qU-X, fa qU-X, fa q)^ fa q), 21 ' ^ 

(9; q) 

X 2 —v 2 

{tq; q) 

vi— Ai (*95 9) A2— vi (?; 9) t-21 
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Proof. It was already demonstrated in Proposition |7| that the expansion of P\ in r„ 
(orp u ) is triangular with respect to the ordering -< (|2.24j) . Applying the operators Hj 
to the equality Pa — X^-<a Pa r v = 0> using ( |4.19| ) and equating to zero the coefficient 
at r u we obtain the set of two equations 

Pa K - h *) + Pa 1 " 1 '' 2 bf~ 1,U2 + Px'^ c ?^ +l = > 3 = ^ 2 . (5.9) 

Solving the above equations with respect to p 1 ^ 1,1/2 and P\' U2+ one obtains the 
recurrence relations for the coefficients p\ 

Pa - I3ZTTPaj (a.iuaj 



g^-Ai-l^ _ g^21 + l)(l _ tq v ^) 

(1 _ gAa-«)(l - tq V2 ~ Xl ) 



As the initial value for the recurrence relations we can use p\ which is extracted 
directly from the expansion (|4.25|) : 

p\ = (_l)A2i g -iA 21 (A 21 -l) t -A 21 ^-A 21 _ ( 5U ^ 

It is easy to see then that the recurrence relations ( 5.10|) together with the initial 
condition (|5.11|) possess the unique solution (|5.6| b) for v -< A. Note that the solution 
is compatible with the triangularity condition v -< A since from (|5.f 0| ) it follows that 
Px = for V\ < Ai or u 2 > X 2 - 

Similarly, for the ^-coefficients we obtain from rx = X^-a R\ Pv the equations 

[a] - h x ., 3 ) Rl + b] BJ{ l+1M + c) Rl M -x = (5.12) 

which give rise to the recurrence relations 

k m + im- (1 _ g A 2l)(1 _ tg A 2l) a > 
™ _ (l-g A2 -")(l-tg A ^) 

Ai,A2— 1 - g 2A 2 -M-2 t 2£2 (1 _ g A 2l)(1 _ fq X 2l) K X ^-^) 

which, in turn, being combined with the initial condition 

R x = (_i)A 2 i g iA 21 (A 21 -i) t A 21 £A 21 ( 514 ) 



produce the formula ( |5.8| b). 

The easiest way to obtain the corresponding formulas for 7r^ and Q\ is to use 
the involution ( 4.16 ). Noting that 

P x = t |A| £ 2 W P A (5.15) 
and using the equality U^p v = t 2vi ^ Vl r v ( p7|) one obtains the formulas 

n u = (t?)M-**fi (5.16) 



20 



and 

which lead directly to the expressions ([5.6|a) and (|5.8|a) 



(5.17) 



Corollary. The matrices p\ and R x , resp. 7T X and Q\, being mutually inverse, we 
obtain the following algebraic identities: 



E ^ = o, E Pa^ = 0, E Qa< = o, E * u xQZ = o- 



li^v^X fi^v-iX fi^v-iX fi^v^X 

if /i -< A and \i ^ A, otherwise i?^ = 1 , = 1- 

The identities can also be verified directly with the help of the g-Vandermond for- 
mula (|2|, formula (1.5.3)): 



2V1 



q,q 



(c; g)„ 



Using the relation ( |4.5| ) together with ( |4.14| ) it is easy to transform the expansions 
( |5.5| ) and (|5.7|) into, respectively, 



u-<X v^X 



and 

where 
and 



p x = Y,QxF„ 

u-<X 



n X A rV > 



¥ x = Y.R v xF v 

u-<X 



R x = R x (P 



(rh-1 



(5.18) 

(5.19) 

(5.20) 
(5.21) 



(for the definition of [i^ and /i^ see fl4.15| )). 

The second expansion in (|5.18|) can also be found in the book [20]. Indeed, 
consider the formula (i) in the Excersize 8.1 from ||20|| , i.e. the expansion of the 
product of two little g-Jacobi polynomials in the form 

p n [xx] a, 6; q) p n {x 2 ; a, b; q) = (-aq) q^ 2 > - — rT - ^ 7 T . x x 



x x x 2 q 



l—m s 



a 



E 



(q- m ,b- 1 q- m ;q) k 



(q,bq;q) r 



(abx\x 2 ) q 



k „k 2 +2k 



k =o (q^q^iq 1 m ,x 2 q 1 m ;q) k 

If we substitute now n = A 2 i, x = t~ 2 y, a = t _1 g _A21 , b = t 2 g~ x and use the formula 
(|2.25| ) for the separated polynomials f x {y) then we arrive to the expansion Q5.18j ) of 
the polynomial F x in terms of the polynomials r u . 
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6. Concluding remarks 



We have demonstrated that even in the simplest case of the A% type Macdonald 
polynomials the method of separation of variables is capable to provide a new in- 



terpretation of the product formula ( |2.8| ) for classical orthogonal polynomials of one 
variable (Theorem |l|) and even to produce a new result: the integral relation 
for the same polynomials (Theorem |2|). 



The SoV approach proposed in |39L 28, 29 1 and in the present paper seems 



to be quite general and could provide thus a useful tool for studying the integral 
relations between various one- and multidimensional special functions. The most 
interesting next application of the method would be the case of the A n Macdonald 
polynomials, n > 2. The work in this direction is in progress. We hope also to 
apply the same method to the pair (A 2 , A%) in the cases of periodic Toda lattice and 
elliptic Calogero-Moser and Ruijsenaars models elsewhere, thereby getting some 
(new) integral relations for the Mathieu and Lame types of functions and for their 
multivariable (and q-) analogs. 
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Appendix A. Classical Ruijsenaars system 

An integrable Hamiltonian system with n degrees of freedom is determined by a 2n- 
dimensional symplectic manifold (phase space) and n independent functions (Hamil- 
tonians) Hj commuting with respect to the Poisson bracket 

{H j ,H k } = 0, j,k = l,...,n. (A.l) 

For our purposes it is more convenient to use Weil canonical variables (T Xj , Xj ) 

{T x .,T Xh } = {xj,x k } = 0, {T Xj ,x k } = -iT x .x k 8 jk , j,k = l,...,n (A.2) 

rather then the traditional coordinates-momenta. 

To find a SoV means then to find a canonical transformation M : (x, T x ) i— > 
(y,T y ), M : Hj \— > Hj such that there exist n relations 

$(y jt T V] ;Hi y \ . . . , H™) = , j = 1, . . . , n, (A.3) 

separating the variables i/j. The most common way to describe a canonical trans- 
formation is the one in terms of its generating function F(y\x). 
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Presently, no algorithm is known of constructing a SoV for any given integrable 
system. Nevertheless, there exists a fairly effective practical recipe based on the 
classical inverse scattering method. A detailed description of the procedure with 
many examples can be found in the review paper |39|] . Here we describe very briefly 



its main steps and apply it afterwards to the classical trigonometric 2-particle Rui- 
jsenaars system. 

A Lax matrix for a given integrable system is a matrix L(u) dependent on a 
"spectral parameter" mgC such that its characteristic polynomial obeys two con- 
ditions 

(i) Poisson involutivity: 

{det(L(u) - v ■ 1), det(L(w) - v • 1)} = , Vu, u, v, v G C; 

(ii) det(L(w) — v ■ 1) generates all integrals of motion Hi . 

A Baker- Akhiezer (BA) function is the eigenvector 

L(u) f(u) = v(u) f(u) (A.4) 

of the Lax matrix L(u), provided that a normalization of the eigenvectors f(u) is 
fixed 

n 

£a,(u) fi(u) = 1 , ( f(u) = (h(u), . . . , f n (u)Y ) . (A.5) 
i=i 

Supposing L{u) to be a rational function in u, the pair (u,v) can be thought of as 
a point of the algebraic spectral curve 

det(L(u) - v ■ 1) = . (A.6) 

The BA function f(u) is then a meromorphic function on the spectral curve. 
The recipe for finding an SoV is simple: 

The separation variables yj are poles of the Baker- Akhiezer function, provided it is 
properly normalized. The corresponding eigenvalues T yj of L(yj), or some functions 
of them, serve as the canonically conjugated variables. 

It is easy to see that the pairs (yj,T y .) thus defined satisfy the separation equa- 
tions ( |A.3j ) for $j = det(L(yj) — T y , ■ 1). The canonicity of the variables (?/j,T % ) 



should be verified independently. No general recipe is known how to guess the 
proper (that is producing canonical variables) normalization for the BA function. 
In many cases the simplest standard normalization a = do = (0, . . . , 0, 1) works. In 
other cases the vector a may depend on the spectral parameter u and the dynamical 
variables (T Xj ,Xj). We shall refer to such normalization as a dynamical one. 



To be more concrete, let = res fi{u) and T y , = v(yj). Then from ( |A.4| )- 

u—yj 



(|A.5|) we have the overdetermined system of n + 1 linear homogeneous equations for 



n components f^ of the vector f^: 



L( yj ) /Ci) = T y /W 
E?=i /i j) = o. 



(A.7) 
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The pair (u,v) = (yj,T Vj ) is thus determined from the condition 



rank 



a[u) 
L(u)-v-l 



n — 1 



(A.8) 



where a is understood as a row- vector. Finally, the condition ( |A.8|) can be rewritten 
as the following vector equation: 



a ■ (L(u) - v ■ iy 







(A.9) 



where the wedge denotes the matrix of cofactors. Eliminating v from (|A.9|) we get 
an algebraic equation for yj 

( a \ 
a ■ L(u) 



0. 



(A.10) 



B(u) = det 

\a ■ L n_1 (u) J 

Also, from equations (|A.9| ) we can get formulas for v in the form 

v = A{u) 

with A(u) being some rational functions of the entries of L(u). 

To validate the choice of normalization a(u) it remains to verify (somehow) 
the canonicity of brackets between the whole set of separation variables, namely: 
between zeros yj of B{u) and their conjugated variables T y . = v(yj) = A(yj). To do 
this final calculation one needs information about Poisson brackets between entries 
of the Lax matrix L(u). 

Now we apply the above scheme to the 2-particle Ruijsenaars system. But, first, 
let us recall the definition of the n-particle system. 

The n-particle (A n _i type) trigonometric Ruijsenaars model [37] is formulated 
in terms of the Weyl canonical system of coordinates (T x .,Xj), \xj\ = 1, T x . e R 
(j — 1, 2, . . . , n) with the Poisson brackets ( A.2|) . The mutually Poisson commuting 
integrals of motion Hi are defined as follows 

/ 



Ha 



E 



JC{l,...,n} 
\J\=i 



n 



is J 
fcS{l,...,n}\J 



Vie J 



where 



Vjk 



t^Xj — t 2Xfc 

Xj Xk 



< t < 1 



(A.ll) 



(A.12) 



t being a parameter of the model. The n x n Lax matrix (or the L-operator) is given 
by the formula 

L(u) = D(u)E(u) (A.13) 

where 



D 



(l-t)(t n -u) 



n 



It— (l-u) \+tj 



jk 



E 



t n + U tXj + Xk 



(A.14) 
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The characteristic polynomial of the matrix L(u) ( |A.13| ) generates the integrals 

(EH 

(-l) n * 2l Tr li (t n - «)(1 - u) n det(z • 1 - L(u)) 

n 

= £(-l)*tV*(t* -u)(l- u)\t n - u) n - k H n _ k z k (A.15) 

k=0 

where it is assumed H Q = 1. 

In |2{J we considered the case n = 3 and constructed a classical (as well as 
quantum) SoV for it, giving explicitely the generating function of the separating 
canonical transform in terms of Euler's dilogarithm function [E5| 



r* fit °° z k 

U 2 {z):=-J o yln(l-t) = Ep- (A-16) 

In the present paper we are concerned with the 2-particle (or the A\) case. Putting 
n = 2 we have, respectively: 

D(u)=^-^ t2 - U U V ^ ° ), Ejk = £±±-**±*. (A.17) 
2*1(1 — «) V Jfe t 2 - M txj-Xk K ' 

The characteristic polynomial takes the form 

t(l-u) det(z ■ 1 - L{u)) = t(l - u) z 2 - £(t - u) H lZ + (t 2 - u) H 2 (A.18) 
where the integrals of motion Hj are given by the formulas 

Hi = T X1 H T X2 , H 2 = T X1 T X2 . (A. 19) 

Xl — x 2 x 2 - Xi 

The standard choice of the normalization a = o?o = (0, 1) leads to a trivial 
SoV in the coordinates x± = (xixf 1 ) 1 ^ 2 (separating the center-of-mass motion). 
There exists, however, another, more complicated SoV. Let us choose the following 
normalization a of the Baker- Akhiezer function 

^ ( t 2 + u tj + Xx t 2 + u tj + x 2 \ . , 

a = (ai ' a2) = ^-^i' (A - 20) 

where £ G C is a free parameter. Notice that the chosen normalization is dynamical 
and dependent on the spectral parameter u. 
Now introduce the functions Aj(u) 

A 1 (u) = L n (u) - ^ L 12 (u) , A 2 (u) = L 22 (u) - ^ L 21 (u) , (A.21) 
a 2 a\ 

A 3 {u)=T X] a 3 {u), (A.22) 

a ( u) - (P-uKS-xMu-xs-j) 
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The separation variables (T yj ,yj), j = 1,2, are defined then by the equations 

T VJ = A 1 (y j ) = A 2 { Vj ) , j = 1,2; (A.24) 

which can be put, alternatively, as four equations for four variables y 1,1/2, T yi ,T m in 
the form 

T Vj = T Xk a k ( yj ), j,k=l,2. (A.25) 

From the invariance of ai(w)/a 2 (w) with respect to the change u — ► tx\X 2 /u^ 2 we 
can deduce the relation 

ViVii 2 = txix 2 . (A. 26) 

Having checked by straightforward calculations the relation 

t(l - y) ai (y) a 2 (y) -t$(t- y) [v 12 a 2 (y) + v 2iai (y) ] + (t 2 - y) = (A.27) 

for Oj's, we obtain at once the separation equation for each pair of (T y , y) variables 

t(l - y) T 2 - - y) R x T y + (t 2 - y) H 2 = . (A.28) 



The (Weyl type) canonical Poisson brackets ( A.2|) for the variables T y . , y^ are easily 



checked on a computer. The other way to establish that the separating transforma- 
tion from (T Xj ,Xj) to (T yj ,yj) is canonical is to give an explicit generating function 
for it which has the simplest form in terms of new (canonical) ±-variables: 

x ± = x\x$* , Tf = T X1 T£ , (A.29) 

y± = yhf\ if = T yi Tyf , (A.30) 

x+ = rky + . (A.31) 
The generating function F{Ty, y_\x + , xJ) has the form 

F{T+ y-\x +1 x-) = ilnT+ ln(r^x + ) + F{y+, y„\x+, x.) , (A.32) 



F = i(C{U-y^,xJ) +C{t*]t H 1 x+,x-)- C(t;t ^ x x+,y. 

-i Li 2 (x^) - i U 2 (xZ 2 ) , (A.33) 

where we have introduced the notation 

C(y; x, y) := \A 2 {vxy) + Li 2 {vxy~ l ) + L\ 2 {yx~ l y) + Li^vx^y' 1 ) . (A. 34) 

Function F(y + , yj\x+, xJ) ( |A.33| ) satisfies the equations 

T+ ~ 

x+d x+ F = iln ^ , x-d x _F = HnT x , y-d y _F = -i lnY y , y+d y+ F = . 



T+ 



(A.35) 
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There is striking similarity between the classical SoV performed above for the 
case n = 2 and the one for n = 3 constructed in ||29|| . Indeed, we recall that 
there we had the generating function F(Ty, y~\x + , xJ) ((2.31) of f29|) of the form 

F := i lnT+ hi(t^x + ) + F where 

— 1 3 

F := i(£(t2 ; y_, xJ) + C(t; x + , x_) — L{p- ; x + , y_) 

-Li 2 (xi) -Li 2 (aT 2 )), (A.36) 

and an extra condition which replaced ( |A.31| ) was as follows: 

x + = t~*y+ , 

while the definition of the x±- variables was a bit different: 

1/2 1/2 -1 1/2 -1/2 

•ju _|_ — ju ^ iX' 2 3 5 — 1 2 

Actually, there is a simple and an elegant explanation of this similarity; moreover, 
in the general case of n degrees of freedom we could say that a SoV for the A n _i 
problem with the standard normalization vector Sq = (0, 0, . . . , 0, 1) implies a SoV 
for the A n _ 2 problem with the non-standard normalization vector a = («i, . . . , a n _i) 
(cf. ( [CTD for n = 3), 



- u u • ari /" 1 + u t£ + x r 



« = | " w _i - 77 ' ~ L > • • • > 1-1 ~ S ' ~ n 1 > ( A - 37 ) 
yt™ — it t4 — x\ t n 1 — m tt, — x n -i J 

if we choose ^ = x n . Let us demonstrate this explicitly. 

If we remove the last (nth) row and the last column from the Lax matrix 
L^ n \u) = L(u) ( |A.13| )-( JA.14| ) then, as one can check easily, we get (n — l)x(n — 1) 
Lax matrix L(u) for an integrable system of n — 1 degrees of freedom which is equiva- 
lent through a simple canonical transformation to the standard (n — l)-particle Rui- 



u 



jsenaars model. This 1-degree-of-freedom-less system with the Lax matrix L*™ - 
directly inherits the non-standard SoV with the normalization ( |A.37j ) from the stan- 
dard one (with the do) for the system with n degrees of freedom. Indeed, to see this, 
it is sufficient to note that separation variables (T y ,y) for both systems (one with 
L( n \u) and one with L(u)) are defined from the intersection of two spectral curves: 

fdet(LW(y)-T tf .l) = 0, , , 

\ det(L(y)-T y -t)=0. [ ^ 66) 

In other words, the condition of the standard SoV for the first problem, 

rank U%r-T B .i) =n - 1 ' (a - 39) 

implies the following condition of SoV for the second problem: 

^KiM-Vi)-"- 2 ' < A - 4 °) 
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where 

( t n + u tx n + X\ t n + u tx n + X n _i 



n— 1 



Applying this to our (A 2 , Ai) pair, we have proved in the standard SoV for the 
A 2 problem (L^ 3 \u) = L(u), n = 3), hence, the same separation variables can work 
also for the related L(u) problem which, in turn, is related to the initial 2-degrees- 
of-freedom (A±) problem (with the Lax matrix L^ 2 '(u) = L(u), n = 2) through the 
following canonical transformation: 

y = t~ 1 y, T Xj =Vv j3 f Xj , £ = x 3 , T v = t* 1 1 ~\„ T y . (A.41) 

£2 — t 2y 

The analogous transformation connects the corresponding quantum A 2 and A\ 
systems. Indeed, let us establish an equivalence of the separating kernel M.^ con- 
structed in the Section 4 for the case of the A\ Macdonald polynomials and the Ai^ 
from [29] for the 3- variable case. Let us write down both kernels: for the 2- variable 

case 

1/(2) ka ( I \ (1 ~ g) (g, q, x 2 , x~ 2 ; g)oo £ q (t; y, r) 

M y ' = M g ,g{r,y\x) = j-j \ -rj r- (A.42) 

2B q (g, g) C q [t*; x, yj £ g (ta ; x, rj 

and for the 3- variable case 

M<» ^ AW. V\x) = {1 ~ q) ^•*y\ C 'V<y> f ) . (A . 43) 

2B q (g,2g) C q {t*;x,y) C q (t;x,r) 



Substituting 

V = V-, 

and, respectively, 



t- l y+ = t- 1 H- 1 x+ , (A.44) 



y — y~ , x = X- , r = t iy + = x + x 3 1 , (A. 45) 

and also 

B ' (a -^ (1 (A - 46) 

we get 

M ( 2 ) = £g(t;y-,t- 1 y + ) (g,t,x 2 _ } xZ 2 ;g)oc (t; g)oo ^ ^ 

(t 2 ',(l)oo 2C q (t%; x_, yJ^j C q (t%; x_, t~i£ -1 x_| X 



m(3) = C q [t*;y-,t ( q ,t,x 2 _,xZ 2 ;g)oo (t 2 ;g)oo ^ (A 4g) 

{t 3 ;q)oo 2C q (t^;x_,y^j C q (t; 



Identifying the variables 

e = x 3 , yj=t Vj , J = 1,2, (A.49) 
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we obtain the equivalence of two operators: 

M (2) = Wy 1 o M (3) o W x (A.50) 

where 

w= feglooA^-^aV) = (t J tx 1 x^ 1 ,tx 2 x^ 1 ;q) 00 (A 51) 
(t 2 ;g)oc C q (t^;x.,t-^- 1 x + ) if, x^ 1 , x 2 x^ 1 - q)^ ' 

= (t 2 ;g)^A,(tj;y_,Hy + ) = (t 2 , y 1; y 2 ; g)^ 

y (t s ;q) oc C g (t;y-,t- 2 y + ) (t 3 , g)oo ' 1 ' J 

In particular, the relations between g-shift operators are as follows (cf. ( |A.41|) ): 

W x o T q>x . o W- 1 = 1 ~ tx ^ f q>x . = t$v j3 f q>Xj , (A.53) 



^ ° o W- 1 = ^—^r f qm = f * * * %m ■ (A.54) 
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